Cross-Validation and Mean-Square Stability

k-fold cross validation is a popular practical method to get a good estimate of the error rate of a learning algorithm. Here, the set of examples is first partitioned into k equal-sized folds. Each fold acts as a test set for evaluating the hypothesis learned on the other k − 1 folds. The average error across the k hypotheses is used as an estimate of the error rate. Although widely used, especially with small values of k (such as 10), the technique has heretofore resisted theoretical analysis. With only sanity-check bounds known, there is not a compelling reason to use the k-fold cross-validation estimate over a simpler holdout estimate. The complications stem from the fact that the k distinct estimates have intricate correlations between them. Conventional wisdom is that the averaging in cross-validation leads to a tighter concentration of the estimate of the error around its mean. In this paper, we show that the conventional wisdom is essentially correct. We analyze the reduction in variance of the gap between the cross-validation estimate and the true error rate, and show that for a large family of stable algorithms, cross-validation achieves a near optimal variance reduction factor of (1+o(1))/k. In these cases the k different estimates are essentially acting independent of each other. To proceed with the analysis, we define a new measure of algorithm stability, called mean-square stability. Meansquare stability is weaker than most stability notions described in the literature, and encompasses a large class of algorithms, namely bounded SVM regression and regularized least-squares regression, among others. For slightly less stable algorithms, such as t-Nearest-Neighbor, we show that cross validation leads to an O(1/k) reduction in the variance of the generalization error.